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MANAGING  THE  DEVELOPMENT  OF 
TECHNOLOGY-BASED  COURSES 

SUCCESS  FAaORS  FROM  EIGHT 
GOVERNMENT  TRAINING  COURSES 


John  Benneftf  CHen  Bunker,  and  Kurt  Rowley 

A  study  was  conducted  to  determine  whether  sucoess  factors  identified  in 
traditionai  higher  education  distance  iearning  research  iiterature  were  important 
to  technoiogy-based  course  deveiopment  efforts  at  Defense  Acquisition 
University  (DAU).  The  study  inciuded  a  literature  review,  a  list  of  candidate 
success  factors  from  the  literature,  data  collected  through  interviews  with  eight 
faculty  course  development  managers,  and  data  analysis  to  correlate  findings 
with  the  research  literature.  The  study  indicates  that  many  of  the  success  factors 
found  in  the  literature  were  also  important  to  management  of  the  DAU  course 
development  projects.  A  number  of  additional  success  factors  identified  were 
important  for  the  DAU  oourses  and  may  be  important  for  other  distance  education 
development  environments.  Recommendations  for  development  managers  of 
distance  education  courses  are  proposed. 


Defense  Acquisition  University 
(DAU)  is  a  corporate  university 
charged  with  training  the  Depart¬ 
ment  of  Defense  (DoD)  acquisition 
workforce.  This  study  investigated  suc¬ 
cess  factors  for  managing  the  develop¬ 
ment  of  eight  technology-hased  courses 
at  DAU  and  identified  success  factors  that 
may  he  relevant  to  ongoing  and  future 
DAU  course  development  efforts.  The 
courses  studied  (see  Table  1)  range  from 
entry-level  courses  taken  on-line  with  no 
required  instructor  interaction  to  higher- 
level  courses  using  sophisticated  threaded 


storylines  and  hybrid  (on-line  and 
classroom)  components.  These  eight 
courses  were  selected  because  they 
are  representative  of  the  courses  that 
won  DAU  two  consecutive  U.S.  Dis¬ 
tance  Learning  Association  (USDLA) 
Awards  for  Excellence  in  Distance 
Learning  Programming  for  2000  and 
2001.  In  addition,  they  have  been  suc¬ 
cessfully  deployed,  meet  or  exceed  all 
educational  and  administrative  re¬ 
quirements,  and  cover  the  range  of  in¬ 
structional  designs  and  delivery  modes 
DAU  uses  in  most  of  its  courses. 


77 


Aiquisition  Review  Quarterly  —  Winter  2003 


Table  1.  DAU  Course  Summaries 


Course 

Students 
per  Year' 

Course 

Hours 

(Online/ 

Classroom)^ 

Development 

Time^ 

Training 
Certification 
Requirements  Met 

Course 

Description 

Fundamentals 
of  Systems 
Acquisition 
Management 
(ACQ101) 

10,500/yr 

25/0 

13  months 
(Oct  97-Nov  98) 

Level  1  DAWIA* **  training 
certification  in  Acquisi¬ 
tion  Management. 
Required  course  for 
multiple  career  fields 

•  Integrated  entry-level 
course  covering  eight 
functional  career 
fields. 

•  Lessons  and  exams 
accessed  via  the 
internet. 

Intermediate 
Systems 
Acquisition 
Course 
(ACQ  201 A/B) 

5,000/yr 

40/36 

18  months 
(Jul  99-Mar  01) 

Level  II  DAWIA*  train¬ 
ing  certification  in 
Acquisition  Manage¬ 
ment.  Required  course 
for  multiple  career 
fields 

•  Intermediate  level 
integrated  course. 

•  Scenario-based 
hybrid  design  using 
both  internet-based 
and  classroom 

(5  days)  delivery 
modes. 

Introduction  to 
Acquisition 
Workforce  Test 
and  Evaluation 
(TST101) 

2,000/yr 

30/0 

18mos 

(Jan  99-  Oct  00) 

Level  1  DAWIA* 
required  training 
certification  for  Test  and 
Evaluation  career  field 

•  Entry  level  course 
with  lessons  and 
exams  accessed  via 
the  internet. 

Basic  Software 
Acquisition 
Management 
(SAM  101) 

700/yr 

19/0 

10  months 

Does  not  provide 

DAWIA*  training  certifi¬ 
cation  (course  targets 
students  from  all 

DAWIA*  career  fields 
and  levels) 

•  Internet-based 
distance  learning 
course  for  all  levels 
of  learners. 

•  Not  a  certification 

course. 

Program 

Manager's 

Tools  Course 
(PMT  250) 

720/year 

56/24 

(Virtual 

Classroom) 

3  months  for 
65%  solution, 

7  months  for 
85%  solution 

Level  II  DAWIA*  training 
certification  in  Program 
Management  career 
field 

•  Unique  hybrid  design. 

•  Internet  based  DL 
(8  lessons)  followed 
by  synchronous  four- 
day  virtual  classroom 
using  phone  confer¬ 
encing  and  LMS  file 
sharing  tool  (Forum). 

'  Students  per  year  figures  are  approximate. 

^  Course  hours  based  on  course  design  estimates. 

^  Development  time  obtained  from  Course  Manager  interviews.  Generally,  from  the  time  the  development  contract  was  awarded 
to  the  start  of  the  first  production  offering. 

*  Defense  Acquisition  Workforce  ImprovementAct  (DAWIA) 

**  Business,  Cost  Estimating,  Funds  Management  (BCF)  functional  area 

Note:  too  series  courses  are  entry-level,  200  series  courses  are  intermediate  level,  and  300  series  course  are  advanced  level. 

(continued) 
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Table  1 .  DAU  Course  Summaries  (continued) 


Course 

Students 
per  Year' 

Course 

Hours 

(online/ 

classroom)^ 

Development 

Time^ 

Training 
certification 
requirements  met 

Course 

Description 

Program 
Management 
Office  Course 
(PMT352) 

700-1 000/yr 
(first  offer¬ 
ings  started 
in  June  02) 

50/232 

18  months  total 
(DL  fielded  in 

12  months) 

Level  III  DAWIA*  training 
certification  in  Program 
Management  career 
field 

•  Hybrid  design. 

•  Integrated  advanced- 
level  course. 

•  10  modules  of 
internet-based 
distance  learning. 

•  12  scenario-based 
exercises  over  six 
weeks  in  the 
classroom. 

•  LMS  used  to  access 
classroom  material 
and  exams. 

Fundamentals 
of  Earned 

Value 

Management 
(BCF  102) 

700/yr 

60/0 

About  20  months 

Level  1  DAWIA*  training 
certification  in  BCF** 
career  field 

Internet  based 
distance  learning 
modules  for  entry- 
level  instruction  in 
Earned  Value 
Management 

Acquisition 
Business 
Management 
(BCF  211) 

170/yr 

20  to  30/37 

About  8  months 

Level  II  DAWIA*  training 
certification  in  BCF** 
career  field 

•  Hybrid  design, 
intermediate  level 

course. 

•  Students  must  pass 
three  tests  online 
within  60-day  window. 

•  Review  material  (no 
structured  lessons) 
provided  online  as 
prerequisite  to  five- 
day  classroom 
portion. 

'  Students  per  year  figures  are  approximate. 

^  Course  hours  based  on  course  design  estimates. 

^  Development  time  obtained  from  Course  Manager  interviews.  Generally,  from  the  time  the  development  contract  was  awarded 
to  the  start  of  the  first  production  offering. 

*  Defense  Acquisition  Workforce  Improvement  Act  (DAWIA) 

**  Business,  Cost  Estimating,  Funds  Management  (BCF)  functional  area 

Note:  1 00  series  courses  are  entry-level,  200  series  courses  are  intermediate  level,  and  300  series  course  are  advanced  level. 


The  process  for  managing  the  devel¬ 
opment  of  these  courses  mirrors  the 
weapons  system  acquisition  process. 
Both  start  with  requirements  generation 
and  progress  through  concept,  design, 


content  development  and  programming, 
testing,  and  deployment.  Interviews  with 
the  eight  course  development  managers 
(all  DAU  faculty  members  with  practitio¬ 
ner  experience  in  systems  acquisitions). 
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confirmed  that  many  of  the  course  de¬ 
velopment  success  factors  are  directly 
related  to  commonly  accepted  systems 
acquisition  and  program  management 
techniques. 

The  distance  education  literature  re¬ 
ports  many  success  factors  related  to 
managing  the  development  of  technol- 
ogy-hased  university-level  courses. 
Those  most  commonly  cited  include 
effective  use  of  chang¬ 
ing  technologies,  suffi¬ 
cient  resources  for 
course  development 
and  operation,  creating 
educationally  sound 
and  engaging  course 
designs,  effective  staff¬ 
ing,  detailed  planning, 
identifying  and  accom¬ 
modating  the  needs  of  instructors  and 
students,  and  ensuring  suffieient  tech¬ 
nical  expertise. 

Much  of  the  existing  information  fo¬ 
cuses  on  technology-hased  traditional 
university  training.  Compared  to  tradi¬ 
tional  universities,  corporate  universi¬ 
ties  face  unique  challenges.  Beeause  the 
corporation  typically  pays  employee 
salaries  while  they  are  students  at  the 
corporate  university,  the  training  must 
he  as  efficient  as  possible.  Also,  the 
training  investment  is  expected  to  trans¬ 
fer  directly  into  joh  performance.  These 
are  strong  incentives  for  creating  effec¬ 
tive  learning  environments  that  mini¬ 
mize  employee  time  in  the  training  en¬ 
vironment. 

The  researchers  used  success  factors 
described  in  distance  learning  research 
to  help  guide  data  collection.  Interviews 
were  conducted  with  the  DAU  course 
development  managers,  followed  by 


"The  researchers 
used  success 
factors  described 
in  distance  iearn- 
ing  research  to 
heip  guide  data 
ceiiectien." 


analysis  of  data.  The  success  factors 
identified  from  the  literature  review 
were  then  compared  with  success  fac¬ 
tors  developed  from  the  DAU  interviews 
to  determine  which  factors  are  common 
to  both  and  which  are  unique  to  DAU. 
A  set  of  proposed  recommendations  for 
distance  learning  program  managers 
was  developed  from  the  results  of  the 
study.  The  study  provided  a  strong 
grouping  of  success  factors  and  recom¬ 
mendations  that  should  apply  to  DAU 
as  well  as  to  management  of  the  devel¬ 
opment  of  distance  learning  courses  at 
similar  institutions. 


Background 


From  the  inception  of  formal  DoD  sys¬ 
tems  acquisition  training  in  1971  until  the 
late  1990s,  students  have  traveled  to  a  DAU 
classroom  location  to  attend  courses.  Stu¬ 
dents  from  more  than  50  miles  away  incur 
temporary  duty  costs  (travel,  lodging,  meals) 
that  are  paid  by  DAU.  Until  recently,  elass 
durations  ranged  from  three  days  to  20 
weeks.  In  response  to  downsizing  and  cost 
concerns  in  the  mid-1990s,  DAU  developed 
a  strategy  to  take  advantage  of  emerging 
technologies  and  join  the  movement  toward 
technology-based  distance  learning  (com¬ 
monly  known  as  E-learning). 

In  the  context  of  this  study,  a  technol¬ 
ogy-based  course  is  one  that  requires  stu¬ 
dents  and  instructors  to  use  the  Internet  and 
computer-based  technologies  to  access  and/ 
or  manage  some  or  all  aspects  of  the  course. 
With  contractor  support,  a  unique  Learning 
Management  System  (LMS)  called  the  Vir¬ 
tual  Campus  was  developed  in  1998  in  an- 
tieipation  of  hosting  requirements  for  the 
to-be-developed  Internet-based  distance 
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learning  courses.  Since  1998,  more  than 
18  technology-based  courses  (commonly 
known  as  on-line  courses)  have  been 
developed  by  DAU,  and  more  are 
planned.  The  number  of  graduates  for 
each  course  ranges  from  several  hundred 
to  more  than  10,500  students  per  year. 
Collectively,  over  20,000  students  gradu¬ 
ate  annually  from  these  courses.  The  shift 
to  on-line  courses  has  significantly  re¬ 
duced  the  time  students  spend  in  the  class¬ 
room  environment,  along  with  the  asso¬ 
ciated  costs. 

Both  mandatory  and  optional  technol¬ 
ogy-based  training  courses  are  offered  to 
over  130,000  Department  of  Defense  ac¬ 
quisition  workforce  personnel  in  1 1  ca¬ 
reer  paths.  Some  courses  are  conversions 
from  classroom  courses,  some  are  new 
courses  designed  specifically  for  the  on¬ 
line  environment,  and  some  utilize  a  com¬ 
bination  of  new  and  existing  material.  The 
first  courses  were  designed  for  Internet- 
based  distance  learning,  with  no  physi¬ 
cal  classroom  required.  Later  course  de¬ 
signs,  known  as  “hybrids,”  included  both 
an  Internet-based  portion  and  an  in-resi- 
dence  classroom  portion.  Table  1  sum¬ 
marizes  the  eight  courses  studied  for  this 
research  project. 

Review  of  Distance 
Learning  Literature 


The  review  identified  success  factors 
critical  to  managing  the  development  of 
technology-based  courses  in  the  tradi¬ 
tional  university  environment.  General 
categories  of  success  factors  related  to 
managing  the  development  of  distance 
education  courses  provided  a  basis  for  the 
development  of  a  research  protocol  for 


interviewing  course  managers  at  DAU. 
The  success  factors  identified  in  the  lit¬ 
erature  review  were  then  compared  with 
success  factors  identified  during  analysis 
of  the  DAU  interview  transcripts. 

Due  to  the  immense 
body  of  knowledge  re¬ 
lated  to  distance  educa¬ 
tion,  selection  criteria  for 
the  search  were  very 
narrow,  focusing  on  re¬ 
ports  of  success  factors 
for  the  management  of 
distance  education  de¬ 
velopment  projects.  The 
search  favored  empiri¬ 
cal  results  from  con¬ 
trolled  studies  where 
possible.  Additional 
sources  that  reflected  collection  of  data 
from  experienced  distance  educators  were 
also  included. 

A  summary  of  common  problems  re¬ 
lated  to  innovation  with  on-line  distance 
learning  (Robinson,  2001)  provided  a  use¬ 
ful  method  for  success  factor  categoriza¬ 
tion.  Based  on  the  experience  of  426  dis¬ 
tance  educators,  Robinson  classified  dis¬ 
tance  education  course  issues  related  to 
innovation,  leading  to  the  four  general  cat¬ 
egories  of  resource  availability,  organi¬ 
zational  issues,  human  resource  capac¬ 
ity,  and  technology  capabilities.  These 
four  categories  were  used  to  organize  suc¬ 
cess  factors  found  in  the  literature. 


Resource  issues  figure  prominently  in  the 
literature.  Among  the  resources  commonly 
identified  as  lacking  during  distance  edu¬ 
cation  course  developments  are  time,  fund¬ 
ing,  personnel,  and  a  sufficient  technology 
support  infrastructure.  Several  authors 


"Beth  mandatory 
and  optional 
technology-based 
training  courses 
are  offered  to 
over  130,000 
Department  of 
Defense  acquisi¬ 
tion  workforce 
personnel  in  1 1 
career  paths." 


Resource  Avaiiabiuty 
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(Alexander,  MacKenzie,  &  Geissinger, 
1998;  Brigham,  1992)  cite  the  importance 
of  instructional  support  services,  as  well 
as  providing  sufficient  working  time  and 
realistic  production  deadlines,  as  success 
factors.  Rohinson  (2001)  points  out  that 
distance  education  projects  are  often 
underfunded  as  well  as  being  too  small  in 
scope  to  he  financially  viable,  suggesting 
that  giving  projects  proper  scope  is  neces¬ 
sary  for  success. 

Organizahonal  Issues 

Organizational  success  factors  are  often 
tightly  interwoven  with  an  organization’s 
structure  and  culture, 
and  may  be  difficult  to 
implement  in  some 
situations,  especially 
when  introducing  inno¬ 
vation  into  the  organiza¬ 
tion.  However,  when  the 
development  team  con¬ 
siders  certain  factors,  dis¬ 
tance  learning  project 
outcomes  can  be  enhanced.  For  example, 
distance  education  courses  require  unique 
internal  coordination  and  administrative 
practices  (Robinson,  2001).  Research 
shows  that  descriptions  of  processes  for 
determining  course  content  and  the  ap¬ 
proval  of  that  content  must  identify  clearly 
the  people  to  be  included  in  the  process 
(Brigham,  1992).  In  addition,  the  organi¬ 
zation  must  provide  appropriate  technical 
support  (Alexander,  MacKenzie,  & 
Geissinger,  1998)  and  consistent  organi¬ 
zation-wide  strategies  for  the  use  of  tech¬ 
nology  in  teaching  and  learning  (Bates, 
2000).  Wagner  (1995)  identified  the  need 
for  adequate  organizational  learner  and  in¬ 
structor  support.  Finally,  consideration  of 
the  overall  attitudes  of  administrators. 


"Human  resource 
capacity  is  lighliy 
intertwined  with 
ether  resource 
and  erganiza- 
tienai  issues." 


faculty,  and  staff  toward  the  use  of  tech¬ 
nology  must  be  part  of  the  course  devel¬ 
opment  (Brigham,  1992;  Volery,  2001). 

Human  Resource  Capaciiy 

Human  resource  capacity  is  tightly  in¬ 
tertwined  with  other  resource  and  organi¬ 
zational  issues.  Several  primary  human  re¬ 
source  issues  were  found  in  the  literature. 
For  example,  development  teams  must 
produce  quality  materials  and  support  the 
instructional  requirements  of  a  distance 
education  environment  (Brigham,  1992). 
Likewise,  it  is  helpful  to  apply  systematic 
and  analytical  methods  of  course  design 
and  development  (Alexander,  MacKenzie, 
&  Geissinger,  1998);  Wagner,  1995).  The 
design  and  development  needs  require 
course  developers  to  go  beyond  general 
conceptual  planning  and  think  through  the 
details  involved  in  a  distance  course 
(Robinson,  2001). 

In  an  analysis  of  104  Australian  tech¬ 
nology-based  learning  projects,  Alex¬ 
ander,  MacKenzie,  and  Geissinger  (1998) 
determined  that  the  instmctional  staff  must 
address  specific  student  needs,  use  the 
technology  to  enhance  learning  in  ways 
not  previously  possible,  use  a  sound  and 
well-integrated  instructional  strategy,  in¬ 
clude  learner  support,  and  design  assess¬ 
ments  appropriate  for  technology-based 
delivery.  A  proper  balance  must  be  present 
between  the  capabilities  of  the  instructional 
staff  and  the  technical  and  instructional 
support  staff  (Volery,  2001),  leading  or¬ 
ganizations  with  less  technical  support 
to  invest  additional  resources  in  staff  de¬ 
velopment. 

Technology  Capabilities 

A  broad  range  of  technological  suc¬ 
cess  factors  are  identified  in  the  literature 
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reviewed.  Availability  of  adequate 
technical  support  is  mentioned  repeat¬ 
edly  and  is  tightly  interwoven  with  the 
resource,  organizational,  and  staffing  is¬ 
sues  described  above.  Lopez  and 
Nagelhout  (1995)  note  three  success  fac¬ 
tors  for  the  use  of  technology  in  on-line 
education:  reliability,  quality,  and  richness. 
Alexander,  MacKenzie,  and  Geissinger 
(1998)  note  numerous  technology  success 
factors  including  software  testing,  software 
development  expertise  (where  relevant), 
copyright  issue  resolution,  and  student  ac¬ 
cess  to  hardware  and  software.  Bates 
(2000)  noted  that  there  is  a  tension  between 
the  need  for  student  technology  access  and 
equity  of  access  to  higher  education.  Bates 
also  made  the  interesting  note  that  due  to 
the  high  and  recurrent  investment  cost  in 
technology,  the  use  of  new  technologies 
may  not  provide  overall  cost  savings. 

The  literature  indicates  that  not  all 
course  developments  end  successfully, 
and  failures  can  often  be  traced  back  to 
poor  understanding  at  some  level  of 
how  to  balance  the  factors  discussed 
above,  or  even  ignorance  of  some  of 
those  factors. 


Data  Collection  and  Analysis 
Methods 


The  research  design  was  largely  qualita¬ 
tive  and  used  guided  interviews  as  the 
primary  means  of  data  collection.  The 
interview  protocol  was  designed  to  fa¬ 
cilitate  the  exploration  of  the  course 
managers’  experiences  and  relate 
them  to  the  general  issues  identified 
in  the  literature  review.  The  interview 
protocol  was  validated  with  an  initial 
interview  conducted  jointly  by  two 


researchers.  A  single  researcher  conducted 
all  remaining  interviews.  All  interviews  were 
recorded  and  transcribed  for  the  analysis. 

The  interview  questions  (see  Appendix) 
were  designed  to  provide  field-based  inputs 
from  the  eight  DAU  developmental  course 
managers  sufficient  to  allow  the  comparison 
of  their  experiences  with  the  success  factors 
identified  in  the  hterature.  The  interview  ques¬ 
tions  were  organized  into  three  groups:  stake¬ 
holder  issues  (organizational  category  issues), 
team-level  issues  in  the  development  pro¬ 
cess  (human  resources  issues),  and  course- 
level  issues  (resource  availabihty  and  tech¬ 
nology  issues). 

The  interview  method 
was  face-to-face  with  fol¬ 
low-on  contact  for  clarifi¬ 
cation.  Data  analysis  in¬ 
cluded  identification  of  99 
independent  issues  in  the 
transcripts  followed  by  or¬ 
ganization  of  those  issues 
into  themes  and  then  into 
candidate  success  factors. 

The  course  managers  re¬ 
viewed  the  results  and  a  post-hoc  analysis 
correlated  the  data  with  results  of  the  litera¬ 
ture  review.  As  part  of  the  post-hoc  analysis, 
the  DAU  course  managers  reviewed  the  ini¬ 
tial  list  of  10  most  commonly  occurring  fac¬ 
tors  and  their  relative  rankings.  Based  on  their 
feedback,  the  initial  list  of  10  was  reduced  to 
eight  by  eliminating  some  redundancies. 


"The  research 
design  was 
largely  qualita¬ 
tive  and  used 
guided  interviews 
as  the  primary 
means  el  data 
collection." 


Results 


The  eight  success  factors,  derived  from 
the  interview  data,  are  described  below: 

1 .  Effectively  blending  technologies  —  This 
success  factor  includes  researching  and 
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analyzing  available  technologies  and 
the  most  efficient  mix  of  technolo¬ 
gies,  and  considering  methods  of 
blending  on-line  and  classroom  de¬ 
livery  methods.  Also  included  is  un¬ 
derstanding  the  impact  of  future 
changes  driven  by  new  technology. 
At  DAU,  entry-level  courses  typically 
require  minimal  instructor  resources, 
while  higher-level  courses  may  have 
more  interactive  designs  (e.g., 
blended  (hybrid)  on-line/classroom 
components),  which  require  more  re¬ 
sources. 

2 .  Technical  configuration  control  —  This 
includes  document  version  control, 
harmonization  of  design  and  develop¬ 
ment  versions,  and  assurance  of  source 
documentation  for  all  materials.  Key 
challenges  of  configuration  control  for 

course  developers  were 
managing,  tracking,  up¬ 
dating  and  documenting 
the  assignment  of  learn¬ 
ing  objectives  to  materi¬ 
als,  creating  test  items 
based  on  the  learning  ob¬ 
jectives  and  course  con¬ 
tent,  and  effectively  man¬ 
aging  developmental  and  production 
courseware  releases.  Real  world  pohcy 
changes  must  also  be  rapidly  incorpo¬ 
rated  into  courses,  even  as  material  is 
being  developed,  which  adds  to  the 
configuration  control  challenge. 

3.  Project  planning  and  management 
techniques  —  This  involves  defining 
overall  course  requirements  and 
learning  objectives  before  develop¬ 
ing  the  course  design;  making  man¬ 
agement  tradeoffs  to  achieve  the 


"Ineffective  de¬ 
sign  strategies  wiii 
net  heid  a 
student's  interest 
in  an  en-iine 
environment." 


optimum  balance  between  cost, 
schedule,  and  quality;  developing 
baselines  and  metrics  for  the  course 
development;  continually  tracking 
and  monitoring  the  course’s  progress 
against  those  baselines;  and  making 
changes  as  necessary  to  ensure  ad¬ 
equate  progress  and  performance. 
These  techniques  are  also  critical  to 
the  success  of  any  systems  acquisi¬ 
tion. 

4.  Meeting  student  needs  with  instruc¬ 
tional  design  strategies  —  This  suc¬ 
cess  factor  is  particularly  critical  for 
distance  learning  or  hybrid  course¬ 
ware.  Ineffective  design  strategies 
will  not  hold  a  student’s  interest  in 
an  on-line  environment.  Data  from 
the  DAU  interviews  show  that  effec¬ 
tive  strategies  include  the  use  of 
problem-based  and  scenario-based 
training  mechanisms  along  with 
storylines  integrated  across  part  or 
all  of  the  course.  Several  courses  em¬ 
ploy  a  highly  blended  strategy  that 
uses  the  on-line  part  of  the  course  to 
prepare  students  to  work  effectively 
in  the  classroom  as  part  of  a  team. 
Other  important  design  factors  in¬ 
clude  planning  adequate  student  time 
for  course  completion,  matching  the 
course  level  with  the  students’  ex¬ 
pected  level  of  preparedness,  and 
providing  efficient  and  usable  re¬ 
sources  to  the  students  (some  of 
these  are  not  directly  controlled  by 
the  course  developer,  but  can  often 
be  indirectly  influenced). 

5 .  Availability  of  Subject  Matter  Experts 
(SMEs)  —  Ideally,  full-time  SMEs  are 
dedicated  to  the  project.  This  was  a 
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big  issue  for  the  ACQ  and  PMT 
courses  since  they  required  SMEs 
from  many  different  departments. 
The  interview  data  indicate  that  there 
was  chronic  under-estimation  of  the 
SME  time  required  in  most  of  the  eight 
courses  studied.  In  several  cases,  the 
orientation  of  faculty  supervisors 
toward  traditional  classroom  instmc- 
tion  made  them  reluctant  to  provide 
adequate  SME  support  for  on-line 
development.  Organizing  faculty 
schedules  to  allow  sufficient  time  for 
SME  support  while  fulfilling  numer¬ 
ous  other  commitments  was  also  a 
constant  challenge. 

6 .  Effective  use  of  testing  and  evalua¬ 
tion  —  This  includes  early  usability 
testing,  periodic  demonstrations  to  or¬ 
ganizational  stakeholders,  formative 
testing  during  design,  and  pilot  test¬ 
ing  by  both  instructors  and  students. 
Some  courses  faced  major  challenges 
because  of  unforeseen  firewall  issues 
and  Internet  access  issues  that  did  not 
show  up  in  testing  and  were  not  ad¬ 
equately  considered  during  the  course 
development.  Early  test  planning  with 
updates  as  necessary  and  feedback 
of  results  into  the  ongoing  develop¬ 
ment  are  important  elements  of  this 
success  factor. 

7 .  Staffing  and  teaming  —  This  includes 
ensuring  a  proper  skill  mix  and  a  sen¬ 
sible  ratio  of  workers  to  supervisors, 
a  positive  and  supportive  work  culture, 
protecting  the  team  from  distractions, 
collocation  of  team  members  where 
possible  (this  included  both  govern¬ 
ment  and  contractor  personnel),  and 
careful  selection  of  working  team 


member  combinations.  Most  DAU 
managers  emphasized  the  importance 
of  an  integrated  team  with  clearly 
defined  and  well-understood  pro¬ 
cesses  for  decisionmaking  and  con¬ 
tent  reviews  (both 
internal  and  exter¬ 
nal).  Early  inputs 
from  all  team  mem¬ 
bers  on  critical  de¬ 
sign  and  content  de¬ 
cisions  and  effec¬ 
tive  and  timely  com¬ 
munication  meth¬ 
ods  were  identified 
as  important  ele¬ 
ments  of  team  pro¬ 
cesses.  Collocation 
of  contractor  and 
Government  personnel,  which  en¬ 
hanced  communications  efficiency, 
was  very  important  for  ACQ  201  and 
both  PMT  courses  due  to  their  tight 
schedules,  complex  storylines,  unique 
designs,  and  integrated  content. 

8 .  Long-term  technology  support  —  This 
success  factor  is  similar  to  the  chal¬ 
lenges  facing  a  program  manager 
when  considering  interoperability  and 
technology  issues  for  a  weapons  sys¬ 
tem  acquisition.  It  includes  long-term 
technology  planning,  consideration  of 
future  requirements  for  interoperability 
of  operating  systems,  ease  of  mainte¬ 
nance,  and  compatibility  of 
courseware  with  future  releases  of 
plug-ins.  These  issues  must  be  initially 
considered  early  in  the  development 
and  revisited  often  as  the  course 
matures.  Several  of  the  DAU  courses 
had  to  be  partly  redeveloped  because 
of  inadequate  technology  support. 


"Early  inputs 
from  all  team 
members  on 
critical  design  and 
content  decisions 
and  effective  and 
timely  communi¬ 
cation  methods 
were  identified 
as  important 
elements  of 
team  processes." 
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For  example,  TST  101  was  technically 
crippled  when  a  new  version  of  Flash 
software  was  widely  distributed  be¬ 
cause  an  incompatible  productivity  tool 
was  used  during  courseware  program¬ 
ming.  Because  some  of  these  early  ex¬ 
periences,  later  courses  used  simpler. 


less  volatile  technology  and  had  fewer 
problems  in  this  area. 

Once  the  eight  DAU  success  factors  were 
identified,  each  was  rated  by  importance  to 
each  of  the  DAU  courses  included  in  the 
study.  A  rating  of  1-5  was  assigned  to  each 


Table  2.  Impact  of  Success  Factors  by  Course 
Success  Factor  Ranking  Matrix 


Success  Factors 

Courses 

Blending 

Technologies 

Configuration 

Control 

Project 

Management 

Instructional 

Design 

SME 

Availability 

Tests 

Evaluation 

Staffing/ 

Training 

Long-term 

Technology* 

Fundamentals  of  Systems 
Acquisition  Management 

3 

5 

5 

4 

4 

5 

4 

4 

Intermediate  Systems 
Acquisition 

5 

4 

5 

5 

4 

4 

5 

3 

Introduction  to  Acquisition 
Workforce  Test  and 

Evaluation 

5 

4 

5 

3 

4 

3 

2 

2 

Basic  Software  Acquisition 
Management 

3 

5 

2 

3 

4 

5 

3 

3 

Program  Manager’s 

Tools  Course 

4 

4 

5 

3 

5 

4 

5 

4 

Program  Management 

Office  Course 

5 

4 

4 

4 

5 

4 

5 

3 

Fundamentals  of  Earned 

Value  Management 

5 

4 

4 

5 

3 

3 

3 

5 

Acquisition  Business 
Management 

5 

4 

4 

4 

2 

3 

3 

3 

Average  Rating 

BIB 

4.1 

3.9 

3.9 

3.9 

3.8 

3.4 

Rank 

n 

3 

4  (tie) 

4  (tie) 

4  (tie) 

5 

6 

*  Values  for  the  long-term  technology  support  success  factor  reflect  the  impact  during  development  of  each  course. 
However,  since  the  interviews  were  conducted,  this  factor  has  become  a  major  issue  for  all  courses  due  to  the  shift  in  DAU 
technology  policy  to  address  firewall,  bandwidth  and  support  issues,  as  well  as  the  move  in  2002  toward  a  new,  standard¬ 
ized,  Shareable  Content  Object  Reference  Module  (SCORM)  compliant  Learning  Management  System. 

KEY:  5-SF  had  major  impact  on  course,  4-SF  had  significant  impact  on  course,  3-SF  had  some  impact  on  course,  2-SF  had 
minimal  impact  on  course,  1-SF  had  no  impact  on  course 
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success  factor  for  each  course  studied,  and 
the  results  were  tabulated.  Table  2  shows 
the  relative  importance  of  each  of  the  suc¬ 
cess  factors  to  each  of  the  eight  courses. 

Analysis  of  the  transcripts  indicated 
that  course-related  factors  such  as  com¬ 
plexity  of  content  and  design,  and  course 
length,  as  well  as  non-course  factors  such 
as  the  background  and  personalities  of 
key  players,  affected  success  factor 
rankings.  Furthermore,  changes  in  envi¬ 
ronmental  elements  during  these  course 
developments  included: 

•  significant  DAU  organizational  stmcture 
changes, 

•  management  personnel  turnover, 

•  increased  LMS  maturity  and  refiabifity, 
and 

•  increased  levels  of  technical  competence 
for  both  faculty  and  students. 

All  these  factors  influenced  our  results  to 
some  degree,  but  those  effects  are  not  spe¬ 
cifically  analyzed  or  addressed  in  this  study. 

Discussion 


The  initial  and  post-hoc  data  analysis 
of  the  information  provided  by  the  eight 
DAU  course  development  managers  con¬ 
firmed  that  the  success  factors  in  the  lit¬ 
erature  were  generally  important  to  the 
DAU  course  developments.  The  most 
noticeable  similarity  between  the  DAU  re¬ 
sults  and  the  literature  reviewed  is  the 
overall  categories  of  success  factors.  Both 
address  issues  related  to  human  resources 
and  technology.  The  literature  focuses  on 
organizational  issues,  while  the  DAU  re¬ 
sults  focus  more  on  program  management 
issues.  This  may  be  an  expected  outcome 


when  one  considers  the  program  manage¬ 
ment  office  (practitioner)  background  of 
the  course  development  managers  and  the 
similarities  of  course  de¬ 
velopment  and  program 
management  processes. 

The  study  also  re¬ 
vealed  some  noteworthy 
differences  between  the 
literature  and  DAU  suc¬ 
cess  factors.  A  focus  on 
financial  resources  was 
less  evident  in  the  DAU 
environment.  One  pos¬ 
sible  reason  for  this  dif¬ 
ference  is  that  the  level  of  funding  avail¬ 
able  to  a  corporate  university  such  as  DAU 
may  be  generally  sufficient  for  the  outlined 
mission  of  the  organization,  whereas  tra¬ 
ditional  universities  may  allocate  fewer  re¬ 
sources  to  their  technology-based  course 
developments.  Another  interesting  differ¬ 
ence  is  the  DAU  emphasis  on  course 
design  and  development  process  issues 
such  as  technical  configuration  control, 
availability  of  SME  time,  the  use  of  an  in¬ 
tegrated  development  team,  and  effectively 
blending  technologies.  While  some  of  the 
same  issues  are  present  in  the  literature, 
they  tend  to  fall  under  the  human  resources 
category,  suggesting  the  possibility  that 
course  design  and  development  at  tradi¬ 
tional  universities  may  differ  from  the  semi¬ 
independent  course  development  project 
teams  common  at  DAU. 


"The  most 
noticeable 
similarity 
between  the 
DAU  results  and 
the  literature 
reviewed  is  the 
overall  categories 
of  success  factors." 


Recommendations  for  Developers  of 
Technology-Based  Education  Courses 


Based  on  the  DAU  interview  results  and 
the  distance  learning  literature,  we  devel¬ 
oped  a  list  of  recommendations  for 
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managers  of  distance  learning  development 
programs.  While  these  recommendations 
are  not  all-inclusive,  they  supplement  and  in 
some  cases  parallel,  recommendations  found 
in  the  literature.  They  are  particularly  apphcahle 
to  DAU,  hut  should  also  constitute  a  relevant 
list  for  course  developers  in  other  environ¬ 
ments  as  well. 

1.  If  you  determine  that  technology- 
hased  delivery  for  part  or  all  of  your 
course  is  appropriate,  do  not  let  a 
specific  technology  drive  the  course 
design.  Instead,  spend  time  up  front 
determining  educational  objectives  and 
developing  an  educationally  sound 
instructional  design.  Follow  this  with 
market  research  to  determine  the  most 
effective  use  of  supporting  technolo¬ 
gies. 

2 .  Allocate  sufficient  resources  for  effec¬ 
tive  configuration  control  throughout 
the  course  development  process.  This 
includes  implementing  processes  for 
managing,  tracking,  updating,  and 
documenting: 

•  allocation  of  learning  ohjectives  to 
content, 

•  assessments  based  on  learning 
objectives  and  course  content, 

•  software  versions,  and 

•  developmental  and  production 
courseware  releases. 

Strong  configuration  control  is  particu¬ 
larly  important  for  courses  with  dy¬ 
namic  content.  Every  team  member 
must  understand  their  role  in  configu¬ 
ration  management  and  dedicate  some 
of  their  time  to  configuration  control. 


3.  Develop  project  management  pro¬ 
cesses  that  support  requirements-based 
development.  These  processes  in¬ 
clude  early  definition  of  learning  ob¬ 
jectives  and  course  requirements 
(prior  to  course  design  activities); 
baseline  tracking  systems  that  allow 
managers  to  track  cost,  schedule,  and 
quality  issues;  regular  reviews  of 
progress;  and  adjustments  to  project 
plans  and  management  objectives 
based  on  these  reviews. 

4.  Use  interactive,  reality-based  instruc¬ 
tional  techniques  such  as  problem-  or 
scenario-based  learning,  team-based 
training,  and  cases  and  stories.  These 
techniques  are  decidedly  more  engag¬ 
ing  to  students  than  traditional  pre¬ 
sentation  approaches,  particularly  in 
a  distance  learning  environment,  and 
can  be  used  with  a  variety  of  blended 
delivery  technologies. 

5.  Allocate  sufficient  subject  matter 
expert  time  for  distance  learning 
development  project.  This  study 
showed  that  the  time  necessary  for 
SME  review  of  materials  is  often 
underestimated,  yet  SME  input  is  criti¬ 
cal  to  the  success  of  any  course.  Eac- 
ulty  SMEs  often  do  not  know  how  to 
provide  effective,  timely  support  for 
technology-based  course  develop¬ 
ments  unless  they  have  experience; 
so  be  prepared  to  train  SMEs  and  plan 
for  a  learning  curve  with  inexperi¬ 
enced  SMEs.  Also,  make  SMEs  inte¬ 
gral  to  the  development  team.  This  is 
especially  important  for  courses  with 
integrated,  multi-subject  material  or 
with  content  that  must  be  integrated 
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across  different  parts  of  the  course 
(such  as  long  storylines). 

6.  When  testing  distance  learning  envi¬ 
ronments,  he  sure  that  the  test  is  con¬ 
ducted  under  conditions  as  close  as 
possible  to  the  actual  production  course 
environment.  Often,  problems  go  un¬ 
detected  during  testing  because  the  sys¬ 
tem  was  not  stressed  to  the  levels  ex¬ 
perienced  by  full  student  loads.  A 
comprehensive  test  plan  is  important 
to  eventual  smooth  operation  of  the 
course.  Also,  test  often  and  at  multiple 
stages  of  the  course  development.  Be 
sure  to  test  both  the  educational  and 
technical  aspects  of  the  course,  using 
both  faculty  and  students. 

7 .  Carefully  control  the  staffing  and  de¬ 
velopment  team  arrangements.  Criti¬ 
cal  issues  include  location  of  team 
members,  managing  distractions  to 
the  team,  and  implementing  inte¬ 
grated  product  team  strategies  that 
allow  all  team  members  to  provide 
inputs  early  in  the  development  pro¬ 
cess.  Effective  multi-modal  commu¬ 
nication  is  essential,  especially  if 
team  members  are  not  co-located. 
Establish  and  promulgate  clear  pro¬ 
cesses  for  ensuring  timely  submis¬ 
sion  and  review  of  materials.  Revisit 
processes  periodically  and  any  time 
performance  metrics  indicate  poor 
results,  and  make  appropriate 
changes  based  on  team  inputs. 

8.  When  making  technology  deci¬ 
sions,  be  sure  to  consider  the  long¬ 
term  viability  of  your  choices. 
Expect  changes  in  technology  and 
availability  of  support,  and  try  to 


be  conservative  in  expectations  of 
the  future  of  new  or  novel  technolo¬ 
gies.  Use  the  experiences  of  previ¬ 
ous  course  developers  to  avoid  pit- 
falls. 


This  study  determined  that  success 
factors  identified  in  the  traditional 
higher  education  distance  learning  lit¬ 
erature  are  relevant  to  managing  the 
development  of  distance  education 
courses  at  DAU.  These  factors  fall  into 
the  categories  of  resource  availability, 
organizational  issues,  human  resource 
capacity,  and  technology  capabilities. 
Eunds  availability  was  one  factor  em¬ 
phasized  in  the  literature,  but  not  as 
evident  in  the  DAU  data.  The  DAU  data 
also  identified  additional  success  fac¬ 
tors  that  were  important  to  the  DAU 
course  managers,  but  were  not  empha¬ 
sized  in  the  in  other  environments. 
These  include  a  focus 
on  technical  configu¬ 
ration  control,  avail¬ 
ability  of  SME  time,  the 
use  of  an  integrated  de¬ 
velopment  team,  and 
effectively  blending 
technologies. 

The  study  suggests  that  the  profes¬ 
sional  education  focus  of  DAU  is  partly 
responsible  for  the  importance  of  the 
additional  success  factors  identified, 
therefore  the  list  of  success  factors  may 
be  applicable  to  institutions  similar  to 
DAU  (in  particular,  corporate  universi¬ 
ties).  Euture  research  should  elaborate 
the  role  of  these  additional  success  fac¬ 
tors  and  clarify  mechanisms  for  their 


Conclusions 


"When  making 
technology  deci¬ 
sions,  he  sure  to 
consider  the  long¬ 
term  viahility  of 
your  choices." 
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application  in  ongoing  distance  edu¬ 
cation  development  projects. 

A  list  of  recommendations  for  man¬ 
agers  of  technology-based  course  de¬ 
velopments  was  created  based  on  the 


DAU  interview  data.  These  recommen¬ 
dations  should  be  applicable  to  institu¬ 
tions  similar  to  DAU,  and  possibly  are 
more  generally  applicable  to  other  en¬ 
vironments. 


90 


Managing  the  Development  of  Te€hnologY-Baseil  Courses 


John  Bennett,  a  retired  Navy  pilot  and  government  program 
manager,  is  a  professor  of  systems  acquisition  management  at  the 
Defense  Acquisition  University’s  Capital  and  Northeast  Campus 
(formerly  known  as  the  Defense  Systems  Management  College)  at 
Fort  Belvoir,  VA.  During  the  past  six  years,  he  has  been  directly 
involved  in  all  phases  of  the  development,  testing,  and  implementation 
of  a  diverse  range  of  technology-based  courses  —  acting  as  subject 
matter  expert,  course  designer,  and  development  manager.  Bennett 
holds  both  bachelor’s  and  master’s  degrees  in  engineering.  Fie  is  a 
designated  DoD  acquisition  professional  and  is  certified  at  DAWIA 
Level  III  in  the  program  management  career  field. 

(E-mail  address:  John.bennett@dau.mil) 


Ellen  Bunker  helped  design  the  classroom  component  of  Defense 
Acquisition  University’s  Intermediate  Systems  Acquisition  (ACQ  201 ) 
course.  She  has  conducted  distance  learning  research,  design,  and 
consulting  activities  for  the  World  Bank,  International  Monetary  Fund, 
United  Educators,  American  Center  for  the  Study  of  Distance 
Education  at  Pennsylvania  State  University,  and  Brigham  Young 
University,  as  well  as  a  number  of  small  organizations.  Bunker  has 
presented  papers  at  several  distance  education  conferences, 
including  the  University  of  Wisconsin  distance  education  conference 
and  the  International  Council  for  Distance  Education  world 
conferences.  Bunker  holds  an  Ed.D.  in  Adult  and  Distance  Education 
and  an  M.A.  in  Teaching  English  to  Speakers  of  Other  Languages 
(TESOL). 

(E-mail  address:  EBunker@isrd.com) 


Kurt  Rowley  is  a  training  systems  researcher  and  developer  in 
Fredericksburg,  VA.  Flis  background  includes  experience  as  a 
systems  consultant,  college  instructor,  software  engineer, 
instructional  designer,  and  researcher.  Rowley  has  led  training- 
research  projects  for  the  U.S.  Air  Force  Research  Laboratory’s 
Fluman  Effectiveness  Directorate,  and  the  U.S.  Office  of  Naval 
Research.  Fie  was  the  lead  instructional  designer  for  both  the  on-line 
and  classroom  portions  of  the  Defense  Acquisition  University’s 
Intermediate  Systems  Acquisition  (ACQ  201)  course.  Rowley  has 
presented  papers  at  numerous  national  and  international  conferences 
and  has  more  than  30  published  scientific  journal  articles  and 
conference  papers.  Fie  holds  a  Ph.D.  in  instructional  systems  and  a 
master’s  degree  in  business  administration. 

(E-mail  address:  KRowley@isrd.com) 


91 


Aiquisition  Review  Quarterly  —  Winter  2003 


Appendix 

Data  Collection  Protocol 


Protocol  Questions  for  Success  Factor  Research  —  These  questions  formed  the 
basis  of  interviewer-interviewee  interactions.  Most  of  the  questions  were  not  asked 
directly,  hut  were  used  as  a  resource  to  help  guide  the  discussion. 

Set  I:  Course-level  decisions 

A.  Describe  how  the  course  development  project  began. 

1 .  Who  made  the  decision  to  include  technology-based  training? 

2.  What/who  influenced  the  decision? 

3.  What  constraints  did  that  place  on  you? 

4.  What  initial  decision  led  to  important  successes  later  in  the  project? 

5.  What  would  have  helped  make  the  process  smoother? 

B.  Describe  the  resources  (time,  budget,  personnel)  available  to  you. 

1 .  Who  determined  the  level  of  resources? 

2.  Who  managed  resources? 

3.  How  appropriate  was  the  level  of  resource  for  the  project? 

4.  What  was  most  successful  about  the  allotment  of  resources? 

5.  What  would  have  helped  make  the  development  work  progress  more 
effectively? 

C.  Describe  the  development  of  the  course  design  and  structure. 

1 .  How  was  the  main  instructional  strategy  chosen?  How  were  learning 
outcomes  (objectives)  selected? 

2.  Who  participated  in  these  decisions? 

3.  How  did  these  decisions  (or  lack  of  decisions)  influence  development 
work? 

4.  What  would  have  made  the  process  better? 

5.  What  worked  best  in  the  process? 

D.  Describe  the  process  for  choosing  the  type  of  technology  (or  mix  of  tech¬ 
nologies). 

1 .  How  was  the  technology(ies)  determined? 

2.  How  did  this  influence  the  development  and  design? 

3.  What  difficulties  did  you  have  with  the  technology  during  the  course 
development  process? 

4.  What  worked  well? 

Set  II:  Team  structure,  function,  and  purpose 

A.  Group  and  team  structure 

1.  Describe  the  team  composition. 

2.  Were  the  team  members  good  followers? 
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3.  Did  team  members  have  adequate  competence  in  their  domain? 

4.  Describe  the  team’s  group  process,  was  the  team  able  to: 

a.  Work  under  ambiguous  conditions 

b.  Tolerate  ill-defined  and  emergent  solutions 

c.  Make  decisions  and  live  with  constraints  regarding  solutions 

d.  Be  flexible  to  changing  staff  levels,  personnel,  schedules,  tool  avail¬ 
ability,  or  other  variables 

5.  How  well  was  the  team  process  supported  during  the  project  (time, 
resources,  organizational  structure)? 

6.  What  technologies  were  used  to  support  the  team  processes? 

7.  How  was  the  team  organized,  and  how  was  that  organization  main¬ 
tained  or  adapted? 

B.  Group  and  team  functions 

1 .  How  was  the  project  vision  shared  with  the  team  (initial  course  design  and 
prototypes)? 

2.  What  was  the  team  involvement  in  definition  of  training  problems? 

3.  What  was  the  role  of  the  team  in  determination  and  updating/maintaining 
of  learning  objectives  throughout  the  development  process? 

4.  What  was  the  level  of  team  involvement  in  selecting  and  designing/ 
organizing  the  instructional  approach? 

5.  What  was  the  level  of  team  involvement  in  selecting  the  technology  and 
media? 

6.  What  was  the  team  involvement  in  the  vendor  selection  process? 

7.  Describe  typical  working  interpersonal  relationships  between  team 
members. 

C.  Group  and  team  objectives,  goals,  or  purpose 

1 .  What  was  the  role  of  the  teaming  arrangement  within  the  greater  organiza¬ 
tion  (why  use  the  teams,  managing  impact  of  new  technologies  on  training 
organizations,  etc.)? 

2.  What  was  the  scope  of  the  teaming  arrangement  within  the  greater  organi¬ 
zation  (involvement  of  upper  management  of  both  vendor  and  government, 
SMEs,  developers,  organization  of  the  project)? 

3.  What  was  the  team  involvement  in  quality  standards  for  all  stages  of 
product  development? 

Set  III:  Stakeholder  roles  and  characteristics 

A.  Describe  the  primary  stakeholders  for  the  government. 

1.  Who  are  they? 

2.  What  requirements  and  constraints  were  delivered  to  you  with  the 
project? 

3.  How  did  senior  management  convey  their  concerns  and  requirements? 

4.  How  did  senior  management  influence  the  work  you  did  on  the  project? 
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B.  Describe  the  learners. 

1 .  Who  are  they?  What  did  you  know  about  them  at  the  start  of  the  project? 

2.  How  did  you  interact  with  them? 

3.  What  role  did  learner  evaluation  play  in  the  project? 

4.  How  did  learners  influence  the  work  you  did  on  the  project? 

C.  Describe  colleagues  (not  mentioned  under  section  two)  that  were  impor¬ 
tant  during  the  development  of  the  project. 

1 .  What  were  their  roles? 

2.  How  did  you  interact  with  them? 

3.  How  did  they  influence  your  work? 
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